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im$mil 7-9XYV-j±<r)>)?ti:<b%>-W>Mz 
*flfct&'J?%< b 1— 9 V b . 

i£ie«<»f y^x-*. 

[ ffig £ 5 ] mil St- ?Xh'J-^ h'^^T'* 6 if 

IBifcTM ^7x-x„ 
•y h££v*. 

buIB*7Vx? Hi. frier-^^ 'J^^-^ y 

awsjf i Kffia^ ^f^7x-x. 

a*. mmf-9x y u-A<7)®^^-7-fe y h 

*il>ft*iI8 iciest y?7i-x. 
[|»5RJi 1 0 3 f-?xb U-Atcrea-r-S.^:3:< b 
t-^XDXyzS* 9 Y Zmfe-f h y 7b s 

Mii-r-s«-^y'^x ^ Mc^-rsfriex-^ a h u - 

[»*^ 1 1 ] ttm-t hmm*y-t v y *&fe?h 
-HtcMji-rs^L/iSinfi^y^x^ hco^-^^aj^ 

v/^^x rf- 97t4 fvz-h&m&m 1 0 c:iam 
im$m 1 3 3 mi*?*;*. 9 yak mdrr-9x y 

tifzjL—rmin 0 b 3T-£>6ff 3<3S l 



?;Hy^^bn-?o*'y Y^<r)9)V-7tt&Zt>\,z 
OfiO^-rcO^-x^'^/M mho 

-9*k mfevmmmmftTJL-wzi: ^xm&zti 

Z>. ffigmi 3iZ%m<r>1im. 
^h^^SE-r&X^-yy-i:, 

SSL-JtHulB^-y^'x ? hlzm^xmi'r-9A Y V 
-Ac7)Jt7-fe -y h ^^SX-r -v Tfc , 

L*r mr ia* 7-t-yh cttjs-r a wie^- wf'J- 

1 7 3 MIBi*5rf 77V. 
^L/Sf[IB7f 7Vx ? Mcwa-ri, 3- H 

*7b y h C7^tXtl l Af7 7*fc % 
2r^tflf *JI 1 6 tclBa^)^. 
[»*«183 mZT9*X-rg>Xr--vTtf. 
mM3-Ylzttm-t&mi7 : -9x h U-Aco-SB^tc 

[tf*ii 1 9 3 buIbs** 1 . http, ftpximoj y 

9—X- y h W3tS*^ -< 7°<7) o 1i>cr>—i£&?t, 
fflBoSPanyti-^*^ >-^— *-y Mc:«M$<Xs '> 
t *>hu1Bt-^X b iJ-AO'>^< fc tlolBSP^ 

[ mim. 2 0 3 KfiES** 4 . mz* yv^ 9 v iz$m 

l»^JS18{ciB«^a. 

(W*«2 1 3 !3iBSffiXT-.yT#. XM)- = y^ 
[ mim 2 2 3 t'-r 1 *ufe ^ U 7 a - 

{cS^v^TBUfBt'^co-^tcT'^-feX-r^ J: 

mtitzmn^uxb. 

[W*ii2 3 3 ltflB*ac-f^M «7>f -V- KX«i 
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m&m 2 2 Kieiww x . 

*JS2 3 [zt^M^JU X. 
2 4(c|E«{<7)x^MX. 

[If 2 6 3 mm&T'U X#\ t*-J» 94 7n 

[it*« 2 7 i Hufe^atT-'^^ x^\ mmmtcomm 
mmizm^n^ wma 2 2 tcaajw>-7W x . 

[19*312 8 3 HfTififflB&^S* 5 , 7^ 
2HZim<T)^UX, 

100013 

{wmmt ss^»3 bx^xiiffljota 

%<r>®.mzm-?h^m-?-*-zmmLxm!k-t& 
ztizmi-z. &m-?-*-tLxiz. y-b. mfa& 
fcco-ttftxii* <v®.mzm-t s»7>ffA zmmx 

[00023 

(LiveBoard) T'^fi^ixS WAfASf^ U (Tivol 
i), Moran, T.P. , Palen, L. , Harrison, S. , Chiu, 
P., Kimber, D. , Minneman, S. , vanMelle, W. , and Ze 
llweger, P. fciSTll get that off the audio": a 
case of study of salvaging multimedia meeting rec 
ords. Proceedings of CHI '97 CM, New York, pp. 202- 

xoizmtztix^h. ^pbzmmiiztihi-tfvco 

[00033 SStBlf SfflcT)? 5X/U— A2000 (Classroo 
m 2000)T1i> 7V-fe>f-y3^7^ r-'OHd&l/ 
$P<#Rft3*i&;&*. t^*ttttJBS*i«rV> (Abowd, 
G.D.,Atkeson, C.G. , Brotherton, J., Enqvist, T., G 
ulley, P., and LeMon, J. Investigating the captur 
e, integration and access problem of ubiquitous co 



mputing in an educational setting, Proceedings of 
the CHI '98 Conference. ACM, New York, pp. 440-44 
7; StXAbowd, G.D., Atkeson, C.G., Feinstein, A., 
Hmelo, C. , Kooper, R., Long, S. , Sawhney, N. , and 
Tani , M., Teaching and learning as multimedia auth 
oring: the classroom 2000 project. Proceedings of 
the ACM Multimedia 796 Conference. ACM, New York, 
pp. lo7-198£#HgOC:i:) . ?7X/P-A200 

i)tfi$&>t>ixh. x^ HJi^-f y#-Yizm*zti. 
x^-r Frfy-uu-vztifcp[)mw£m^x;-hft 

mm LK&mz 1 ~> x tstwutt htitz^mfx ? a 

[00043 ttzmcr>mX'ii. 7 * A (Forum) ( Isa 
acs, E.A. , Morris, T., and Rodriguez, T.K. A forum 

for supporting interactive presentations to distr 
ibuted audiences. Proceedings of CSCW '94. ACM, Ne 
w York, pp. 405-416£#tBtr, < l£ : )li, jmLtzTU-V 

nf-ya >0>mzmix^&. ftfe<7>&j%x'A74 

K vt -7- 9 \z J; -5 X X y 4 Y dzmi&tm h ZthX'% 

[00053 W\<7>mX\Z. XbO —AX (STREAMS) (Cru 
z, G., and Hill. R. Capturing and playing multimed 
ia events with STREAMS. Proceedings of the ACM Mul 
timedia '94 Conference. ACM, New York, pp. 193-200 

yu-eyy—i' a >W&<r>tz#><r>i'X :: rJ±X'$>& „ zti 

vnm-*mitth*LMzi>®m-h. znirm*. m& 
^vsu*^ < X7H ttwrnz-rz, t^o ran*>'£> 

■S. T*XY<r)-3*yY*ftm-thZk\<z£'>X*)V3. 
-*pl,zm.m Ltz b'^^-X h U -A tci£8?£ -o»h z\k\± 
XZhtiK 7Hr>f-y a>tt><r)S- b^mt^- 

4 7Mt7Hfyf-y a y*^^ J- Y^m&ttz 

[0 006 3 $^.tc, x^yH7o-y>fyWt^ 

y-hf^RK^X^A^Mo^^n-Cv^. mx.it. FX 
PAL Dynomite (Wilcox, L.D., Schilit, B.N., and Saw 
hney, N. Dynomite: A Dynamically Organized Ink and 

Audio Notebook. Proceedings of CHI '97. ACM, New 
York, pp. 186-193Sr#Bg<7)Cli:); RX/^m^izmi 
(%mT&*— 7 s 4*; — X-~f v? (Audio Notebook) (St 
ifelman, L. The Audio Notebook: Paper and Pen Inte 
raction with Structured Speech. Ph.D. Thesis. MIT, 

19975r#B^c7)^ b ) WWfbtlt . 7t4 o^-v -y b 
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(Filochat) (Whittaker, S. , Hyland, P., and Wiley, 
M. Filochat: handwritten notes provide access to r 
ecorded conversations. Proceedings of CHI '94. AC 
M, New York, pp. 271-276£#Bg?).r b ){±. 

PCayVz.-fX'&h. £*Z. A(NoTirae) (La 

mining, M. , and Newman, W. Activity-based informati 
on technology in support of personal memory. Techn 
ical Report EPC-1991-103, Rank Xerox, EuroPARC,199 

i) a. xht3-t> zmmLtzmjzxiz 

[0 0 0 7] Mf^^Xf-A^^ixX^h. V- 
3f — (Marquee) (Weber, K., and Poon, A. Marquee: a 
tool for real-time video logging. Proceedings of C 
HI '94. ACM, New York, pp. 58-64&#H&7)>r b)kt. 

W*^. V— 4r-0±»)*Uv<— ya^tt, Wher 
eWereWeV/l/^ f-f7 v'.Xf £>C0t JU-b'-r :* 

ft"CV^-£>(Minneman, S. , Harrison, S. , Jassen, B., K 
urtenbach, G. , Moran, T., Smith, I., and van Mell 
e, B. A confederation of tools for capturing and a 
ccessing collaborative activity. Proceedings of th 
e ACM Multimedia '95 Conference. ACM, New York, p 
P. 523-534§r#fMc7).ri:h TV-HVanna) (Harrison, 
B., Baecker, R.M. Designing video annotation and a 
nalysis systems, Graphics Interface '92. Morgan-Ka 
ufmann, pp. 157-166$- #BB<7)>I b )&f r EVA(MacKay , W. 
E. EVA: An experimental video annotator for symbol 
ic analysis of video data, SIGCH1 Bulletin, 21(2), 

68-71. 1989. ACM Press £ #K<7) £ i; ) f±, f^b 
<-^«yXfAfi>S. t*-r*y— 9 — (VideoNoter) 
(Trigg, R. Computer support for transcribing recor 
ded activity, SIGCHI Bulletin, 21(2), 68-71. 1989. 

ACM Press$-#f^cOifc ) ti. T9^ta h*f ■< ( t*7* 

X (Sharp Zaurus) (Zaurus Operation Manual. SharpCo 
rporation, 19%) HMShTh D , TrVftMlt'yWVS^ 

lzvy?Z-£Z>z\b&X'Z&. 

[0008] 

flRW 1 0 2 S:*^*^b*T^S^K 1 0 0 



Tttm&wimz^i-. s±i i5. mzmti 2 

0. ^i*>9 12 5. 3TML1 3 0, a-?m 91 3 5W 
Z istsWm V C R 3 y h D-;P*^tt £>fi & £ b T-jl- 
-9->'b'-f5j-^-fb>'-hT'#^„ L*>U b'x^-rt^ 
5£<0&S£&ttS-f.?>i§£. #tcy-hx«b*x^-tcBga 

[0009] 

r-fftfc-fe -yS^ a y (S— x-f^. THryf-yj^ 
XtifficovSIS) ^fttftCi-if^df-y^-ASrStRL 

i-u y- r-tm-fe y ^ 3 ycob'^xtiy- h 

S*t£. Sx^Lit^-7^-AS.r>'aM±»^ttfc^ 

^-y-y h^-7i/- ^^zmtxzmm-hrvy 

oT^lr+ttSftftb'f^Xtex-?;*. r- y -J±<7)mm 

[ooio] z\tit>ff)mfm0)mm±. -r^^b'-f * 
mf4y7~zj—y*ife$L-$-hi'X ; ri± (/-r-n-y? 

[0011] x-^x h >J-Act)^< t i-SfcJH:** 

iznmt&f-fx v u-aco— mft*^-? 

s Jjl? blzffi&t&z-FZ&ttiiijjZ'&tJ, f—9X 
h V-MZT7*.X-t&fzlibC04 >?y*-XX'fo&. 
[0012] y-^x h U-A|c»W-«iJ«ar< k 

yb -y hs^js-r&wa^y^'x^ 

[0013] x-^x h v-Mznmt 4*/yx^ h 

y-A<59^-7-b y bSr^L. ^gU^7t7M:*} 

[0014] tr-7 f **gifc:«B«i*n/£^u 73-Hi 
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[00l5]it. #3S H JfoL y-bWforJMx^<r> 
1S^<07-'-^^h 'J-acoA***, «fc«Htifc«>(: 

S<07*— **HJ-A (Clc7)J;5^7-'-^XhU-A 
<±, ^>"<-XA:*K b'r^A*, «StOb'x^A7J. 
x-*XMI-A. *<7-< h^-HA^tfc'^^tfffiS 
OJL-ifATjS-^-tf) lcra^5l±f?^Wt-i.-fc^T 

?XMJ-A<?)A716D3*>tf) — 7 l^-A£jMtR 
L> &8?Xtea^£t§fftLT^IttH"U teiWLttlHK 

[0016] rufyf-y 3 yxtex-^JKm^S* 
m r )mzti&. tm. wmRVttMLtzT-?rvyii 

(CIS? ftTfiM? L £3ftXfcti£R0?I£ *1ktb& frXti 

l < imtR&xmm-t & r v y kuw-t * > 

ti. ffJnSfufc (XJiffilco) t'WXH. TmLtzfO 

yimm?hmmms/xiiimiz£'iTftmztiti7 : 

—9X b U-^&M^^^Xiii&^—fX F y 

[0017] #&hh£ i t ars . - 

Wet -5tf-^x h 'J-A*<K#?ii, Jf-7l/-A 
XJifflJ^-^StRSr^-r 6 i fc J: o T Sttftfc:£ 

&s#u z^-yu-^ximo^-fmum&z 

(THHfUZ^ttiX^X^X^K 
[00 1 8] *<y F7-:5\ «itff^-fx 

a J t£8?X«i/- F M^ftt^ yo— FX(2:/o- 
b'r*x F 0 -A $rJDtf#^6 >f > -y F 

h?Vy*^hmi!J)\i3--y ! \ l z£~>X%&Ztl. ? 
0 ytehttpXliV-^-'\<r>mcr>*>y F 7-7fflmT F 

ft£&SA^b'-f:*£W±U PC. ;\-AfAM^, 
->iy-rl^b'^-xX{iffic7)7-*-f xyuj **-XMZ 

[0019] mixtey y > b^-wmmzwrn-rz 



wwmzti... rvynTY^xt-^vVtfmmz? 
a a 5^ yizffl&Lx&mttti. tsmzvvtyrzti 

n<?>?>u x<rmm4 yfy^-xxtev^yy^w 

9Ay<r>7n?? y?y x—xizMQZt 

h V?*T? b-X ttzT 9 *x h . 

[0020] 

[002 1] |S|-#M#^|5l-X(SW^&^Sr^ 

y-hft5«rr'j^-^ 3 >'2 i ozmr-tz^y 
K-x<v3y\ZjL—?20 0tfmniZtix^&. y—b 

2 1 5*#0. T7t- ^7*b*7-'^^&7t«)<7)/jN$^rb* 
■rsT^-f >F>>2 2 0*iJ>*. r^f^t'f^liT^ 

-a£ jl— vyatry oy-^u y^zm^t 

/-hMr7Uy--ya>'(t b'-f 

fc7l/-A (i^Utf. 2 25) Sr^-O-^fllcOV-^y 
KieSLfcO. 7^-A£A£3rir»Btfci LTb-yh 

3 0<0 r^y^j xhn-^iBX#$ix-So ^A*^ 

l^mmmtJ y 7 xbv-?lzte? J £>x?yy 
**ttW&*u b'rsi-XJiffi^T-^^ h 'J-aco^|& 

[0 0 2 2] r^f^t'f^tt, *fe«9>6V-^*^ 

t t y- h wm-y v r- ^ a y izta -h $ ti h a t wq 
b'^y-xji. a^co*&r. Mitxmft*}* 

vXlZF^jLtybtityfrC 'jrTuy'i?^-, 
f-Wb\ ^-y\'-^..y VX=74 b~cr>V* x.Xyb#X 
7. VC RXiZ&M?) b'x^-X h 'J - A>Ktf) ^ - 7 r*» 

Xit^yzyt'j.-fizmmtstitz'bZtt-fy 
^ y -xifj - b f^^x^A^ajT) $ tit . 

[0 0 2 3] a3St/4Ji. b'r'^^ ^H>73 2 0S 
Vt'f*3yhD-/V3 1 OS-S-fry-Hfrfci^xA 
OX^ 'J ->-j^ 3 ••/ b 3 0 0 ZM^-tZ . b*-r^»> >f y 
F •> <0WK« b'f * i; WIST § »O=-x0 ^ 

^y322) li^-A-v^^jgU >fiaw^y3 2 
4(±HfA^-<;KCaj^tf. -tA^^;U3 70) i&y-h 
^- 3 15 <?5v- ifyizmm L . -#T<?)t^ v 
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3 2 6(4. *SfcH*3 9 0 (EP*>. &mzmm?&fr 

mvm&fzwztfckUcfr^Lti^wigt) *y-b 

^-y«W3 8 0*=aHW*. 

[0024] t'Tfr-i >W3 20<7)Zt>iZZEmi±V 
CRlnyhn-/l'3 6 0(0'fe7l>'eJ)5. 94J*y<i 
V3 5 0(i. S£3*lTV^b7 r :*<0&£cOI$fS^£ 
^-r^ >"?3 5 5 2r*-r6„ #4 y- 

XbU-A b'x^A^J, b*-b'A}> 

2W^3.— - fj^^tfO^y^— XA:*J«D** ) cot^fSegi, 

«Wt4< icHffl L t y # ^XAXJiSiJ 

coi&jfSpy ^-XA £JHi *T J: \>\ 
[0 0 2 5] ^iOyXfACiot, JL-^i-f^'^^ 

X^t^XiiT—T. C Dlfcoj: 3*ft<WW Atcfc 
*ft£ftT^&) b^^jl&fr'^y^-xcon^b.x 
-?T-y-b£Bl&C:fc#*T'£.5. S-hlzl±?J A.X 

b7 f **^c01i?itB&£. v-S^©***-^ 
hLTXiiy-b^-^'coy/i^MXcoHmi: LT. y 
-bWt8H-&£*:#T'£&o ^h.fcOSfltfcfc*^ 
x ? yy mUf *>ti. £tzt'T*i,zmM2tvr;^& fz 

[0026] b#j£-t>y zsb ><r>m. 
*ZW>k-t&tzMz;- bffr£>-x^A £&mi-& £ t 

^xbo-^&t/y-bcotfihBmK.fco-c^ 

f&ztitim^z&m lt b*x^co#^si5^r ^ -bx 
-r & £ t **t# s . j-— nmrn^m^y^^ ?h£ 

(Double Clickm£r>£o%m<rm!iiJ<#~XJ>.$:m^ 

[0027] ±lE<0y- hM^^fAli, FX Palo Al 
to LaboratoryT'il^Lfcy — Y)V »/9 (NoteLook) i^X 

y-Mz^x&mzwRLx^h. *m\*mmt 

X-J-yT. mi%k* : $:W.fti-&tiMZS-h>l'V? 
AVftWS- hMy^fA fcfiJffl-r-S £ #T£ h . 

■&m&mthtz#>\z. fx*, ?wf^ 

fl&cO^aSr-^K'- b-r£3Kc0ffec0;yM TW^ x£ 
££**?&*. <-£S&co$i. &M<Of470)f t 

-?x h o-AZJmx-z&T'V x&TS*mi<vi3m 
Ris/xtewmzmm-f&fztfxD (?>u xi,z$£ti& 

[0028] #i#t©£85ffr?-|>*:i6tc:7 U-Ab— bfiE 



Afcfctt&^wwy- h^-f^Mx^aii^ixs. 
^f^rxh'j - its -7T- D r/bco^s 
Ocoy 1/-AI/- bffitg£^2-t!:&£ i: *>T# h 
( IB-fe . 30f pscO h*7-'* ^ a >y b S: 1 Xti 2 f ps fc»J?3 
-£TiJ:U) . CHUrio-ty-bA-y^co^-fTV 

y- hft*5*'<4 xT'co^&b'r^iSy- bfftfc 
MStcfc V JiSMHTtt* < , HIS s y - b f^fiK^'b' 
T^tcSi« LT l^StfiHT'tt ( y- hf^flJi^JiM*"*^ 

V">L. ff^/f- -A{4giJc0-9"-^'- tlXft 
X(i*l^$ixT^iV\ 

[0029] y-hffr£^xxA(±. ^rt^y^st/y 

oy'i ^ 9-frt><Wf*£ KXJiV-f -\-i^x 

* y h v-7 lt y- y ^ n y b £SiM 
T'# S ^a^OlS^T-!W{c*fflT'J) £c0i5^y- 
hfl«^^ XS-^ST-fflffl-T 5 £ fc {ij5im(c«mT- 

izmirt'&zut&t r>mmx°z*w lkw->x. & 
mmtrt's- h&v : &mLtzXiT*izT?-tx~?&tztb 

(VM<0¥-&£x-—ri l z5-z. hZb -hmmx-fo h. Zcofz 
ib<7>mhmmcJii&<r>-' 3-—V t tZS-h<D7V>' 
hT*7h*Wm-?Z>l\bX']b&» L*»L. ZtvtSiXM 
mm Liz tWW ^-bXfi-^x. ^ix^rvv 
[0030] ^HJJi. y- h . ? u y y , x^- 
-/r^ 3 -y h (m^mz;-bt%Fth) <r>^—97v 
7^, y - b ^-t v ^ a v<r>wt* izmm l tz b'f^f 
—fx v o -j±^<r>T7-<e.xzm.m-hm<r>~rv >yt 
v bxtiffecoaj^^a^ao. y u 7iit'f^» 
^y-fe y b^tiffi-rs^, b^to^^^tr^ 

*zx-$%c\klrx%h. $^tc. ^yytid^^coy- 
b tB^-f mm Ltz bv-'^coffiS^^-fb^-g. £ i: *< 

[003 1] Sfcti'^b'x^cOT^bX^Sflt-ri,^ 

vj-t*i. 7'jyh (c9 LJt b'r 3-9"? 'J -co— o 
CO 9 4 ?\$ . 1999^3^ 120 (ctBli Lfc *S#-^m09/27 
6.529-f-CO"AUT0MATIC VIDEOSUMMARIZATION USING A MEA 
SURE OF SHOT IMPORTANCE AND A FRAME-PACKING MET HO 
D"tV^^>f b;bC0UchihashH;J:4(^T#Mfc LT* 

msm^mm^tii,)mmjm(^r > j~x'Si?,. z 
zx\ t^mt&mmzftmzti. *h xxs&iit&t* 

-7U-J*tfy-? , J-i:LXMtRZi\&<. 04«, t'f 
j-WV-^mZ^-f. At'^-yW-Aiibr^co 

•r 4 »/- x y^-^'tcifv -^naf 4 £ t *>'t§ 4 . 
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[0032] hzmrnommx-iz. w v-itmb&t 

titzfV? tMzTV yhZtih. ZixlzkiX. *7x 
7±.cr>ziyv i-rxM V?y ^-XZftl,XX'\±%< . 

FX^+tJ-OTUr'JyMt t'-r "J - COM 

Ufca5»*=a»a*ii«:^ y 7 *j£**- & -r a #-es 

6„ X^A-:M±^x7f Ub'(WebTV)<Oj; dSrf^ X 

xtiSStf)Pa;:fgii£*u oiix^js^* Lfc^y 7 i 

?fe.X^-&. 

[0033]ry>h?fLJty- hxti br^-fl-v y - 
tca^^iutx-^^'y^tioT. 3--- imt-r* 

•fe y hXtiflfeOx-^T-hy-A^^-rSit^Griix 

[0034] H5 Ji^UBJcOx^M X?:^-^, . Rffi 
fAM^4 0 0(i. *7-f h*-FX<i 

-^h'J-A4 lOJitj. J--ifA^j4 2 0{i, ft 
mOT f -^A*Xby-A4 1 0O»#*»S£U Xft 

[0035] Rffr7><<f 74 0 0(iTO#/jMf?^*-X 
A4 2 5 5r#^, ^iO^^-XAIi, ATJX-?* HJ 
-A 4 1 0tf0JL-^fS*RgS^StXJt--if*W-t7taS?X 

Wt, ^/i^> frtttt?'t* ^430 izfemtz . vmr 
;^^4 00li, AftT-fxhO-^iffltontztblz 

X4 3 0tctll^LTtJ:V^L. *tc-9w\'-T«*f LT 
*>«fcV>. ) 

[0 0 3 6] ^-?y-^^— XA4 4 0(4IX#U)^X-t- 

"/Ty 3 7b, as?. y-hzmzL. ztiatujiiz 

-r'U x 4 4 0 ii . L fcJ&Wfcft — 3Ja±<0A 
^f-^^h'J-A4 1 Oco^r- 7^-^S^5-S*L 
f#S. tKTjti. ^-y-h/W3^t^-^4 5 0f)75'> 
■9*7^4 7<oJ; -5 3r*7x /f -f X7H ffl{:7 v-y 
n-Katr-jB* <«) 4 5 5tc7y>-b 

WSW) fa?) 4 6 0^tS#j$ixS„ EfJTJti. 



[0037] B6Mt*%91a^&0f**-£ . Xf <•/ 7 
5 0 0T<i. 7-hM^fAi ! X^-ht§. J— 

t>ti. SIR $ iifc x - * 7 MJ - a £ £ l fix S * 
JyWU XX'$>&. Xf'77'5 1 Oty-hMy 

xji&s?eo8^i*-t-£ txo izmzmif u y- 

X h y -A LTixJRS-f+ftS i i>T$ h . 
[0038] ^f'v753 0X\ afJXtijgiRTHvfft 

^#)rrsffl#tftfc^y7^ja^>a±^ /'J7*«i 

h) . X^>y7-5 4 0T\ 'J 7ft 

[0039] j; d *i^oga^ 

^y7{±, w^v/x\mw,znm-h?-9xY*) 

[0040]yy7<±. (#^<7)7-hfe-y K^j£-r 
£ ) -9--^'±OWa-t^»b'7 i ^{4S^)fflgX{±'?-c7)t' 

kSK«Ssn.Tv»4 t^tcy yft&fMz. x«± 
-?-o i: & miiffllf to fix v *» 3 y v b $ *rc v ^ft x 

mizy v y * u >- ? $ it s ^ y 7 s-^ffl-r s o*< 

7*»7 >f > K <7Xti7- f#fR7-V W XtCj; o TS^Stl 
&m<0**;-XA,l l z£'?X) b'T^ffiBSr^^Tit 

[004 1 ] 7. h'^KX^ 

£tefflLT«£^y-M;:aj:>rf&i:. x— Tli«S 
i.^. ^Smra«0Jj-7fe>yhTb'f^S*S:^ 



(8) M2 002-57981 (P2002-57 9JL 



[ 0 0 4 2 ] 7^®«OHSS<7)®JiS*^BBTf IJffl 

nftjjo&m^mz 9 v ten lx to -v 9 w 
y*<imt&. I7^{:i±, J-vftmrr'^xn^ 
-^'±tc, *ri±*4i\> (.xi-^r) 6 i otwa^s^" 

U76 1 >9Xbn-9 (4 >9Xbn—96 

2 0. 63 ORX/9'0 7625. 635) cO&ry'/U— 7° 
* 5 ^§ixT^S. X7~Jx?h JURIST >9 

06c7)5agX{±Cixt|Hl^<7)3!!US^J: 
o-c^lttttU #;/y 7 letting. fi!i?);M r?)* 
x 7 h *> BJ*t=*§ if+ft t j: < . immm 

■§StsM&Xi±7- f&istitrv y&Wn ZtlX 

[0043] h^mmmmx-ii, *y~jx. 9 bkzm& 

b*x:*&t**7b y h Jt^ts^xy h y — #4 

[0044] 4v-7xbu-?um±mmi-&4>-7x 

<±*w»o«asirt-C'*4. «*tr, tiixautcj:?^ 

mu^7/7 7^/^-rfl:n) . J&filgiSfcJ: 
■g> ^-T'-rt: ( *gg $• tmw- 3 ) xttumfc: i s 
r;l-7tt.ifimft>ti&. LfrL. s<>xbv-9l±m 

y<r>xba~9^mm^.ti i A^^. Ltzifi^X. m\ 

[0045] C:cOTif -f *HU t'x^VU-XliS 

[0046] H8Ji. b'^^S^L^U^Srffi^JitJ 
4. X^-y:T7 0 0T"t*x*£^rt£. b'-f'^i, 7 

-hftfS.?;^ AizxJjZtLtzt'7*x h y-A-e£-? 
TiJcv^U S—x-f >7\ vSi&X<ifl«i(7M^y 
KCWSI LTffl^rt^HL^ffiScObT 1 ^ h y-AT 

fl!!^;M7V)Tn^7h£S>ftU?;gittttLT 

(mm. w») r-r ay. /?7X 



[0047]Xf7771 Ot^LtbT^A^^f- 

& i t **cs . si^/x{±?hgps^<7)-fe y h zmm L-x 

*-y]s-J±*M$tthZti>X*h. 777 2 0 

gift<t u , y y y izmmm 

6. <fe^-r-?(±, t'fJtXttf-^^b'J-i., htt 
P. ftpXHTGLO-T—fX h y-A**r?-feXT-# &ffi! 

T^y7S:«r*MtLT<)J:V^ (BP*>, X-r>y 7°7 2 0<2 
[0048] giWWcJDtfiSft (m*-i£. $&i 

b'T^v-a y h (Rtfcf. *V-f h^-Ff-^h'J 

-A<ox5>f Hsee> *a«wfc»B&r*fcAfc:. sis 
i^m^. m9<?>mitfy~?v-iz*kzti&x7-< h 

^S8 1 OS.1^2 5 : Q0c7)tt!fc-fZ>74A ; 7-<i>X1- 
•y7y 3 7h86 0^J;at:. ^ >f yfc 'J >? $ 

[0049]Xf7773 0T'{±. fficoa«?X{iMtRt 

nm-hyvytffflSiZtih. x^-^y 

74 0t(l (^J^.tt\ St) tf£j$.ZtL. ZWiiJ) 

-9X h y -A£#B?. 0 Bf V til L*: 0^-4 . 
[0050] flfecoHM<7)»®T{±. ffi^aiix^x-^ 

fpji.«f. yyyrYvxij-^Yte. TV>bZ 

iifzS-b ^- i^'XJi^v 'J -OT^^T^ ^ A 5 A 
yt,ZWMZtl&. H9Ji. /l)7TH^-^7h8 
2 ORXmm-tZ 94 A5-f V8 4 0 $r*-ri,b'x^-9- 
Vy-8 0 0£i5«ifrt6. ?^fA5^y8 4 0{i*ft 
BS-H^ratc-? >y b>^47ttf)tcffiffl$ii. jT'J 77 F 

i^x^-^>y f 8 2 o cottmi-zimizmBxiiimB 
iz&^itz tix \zT*izt5\^xmmtt»zi\x v 

?hXtt (-9-vy— j6»4)<0) ^-7^-A^i,*^* 1 

(^^A.-f. b'7-'3Tcr)ffS<o^>-Mzr?-fex-rsc: 

[005 1 ] SSiffltLT. tt'r'-yay^ilW^ 

(b'x^-9-7y-x{±y-h*^co) &n<v*y"is 
^9b^9^M.y^ y±(omwj:{mi l zm^ lx i> * 
k\ H9{:<ia^<7)-9"7y— *7iSx.7b (*—yu— 

A ) 8 6 0 tf^Ztl. bT-'^-cOD"? 'J -^■7'^x 9Y<r> 
fiStC^JS-rSi/y 77 Kl/X#-^7 h 8 2 Oc75## 

[0052] -f'-^y 7^-^ y hcOESii. a.— 
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mtV Z & X 3 lz 9 A A 5 A vldHiSfW 3 £ tfiK <fc 
I , * -f A 5 A XtiteS#ji56 
3ix& t -?-?);5M y±^^/y'x ? hX»2{2B(;: 

77 H VXf]—^ 7 b820 cOfitBfiHI 9 iZ^tX d IZ 
94 ^A yX*)±X$>~?Xi>X\^L. ?4J*"74><r> 

[0053] mmxi$imB?4J±z.ir-)v*m^~tz> 

tiMz. fJ^AyXiZyVyTFl-Xtl-^yhiz 

mtRV/xitm&WMz tix^xw m-coxim®. 
w& ( mm<r>x\m%h ) * * ax^-zpt'^sc^ t'-f 

#&«vn"7 Wl-7 FW*-^7 hXHJ 77S:i 

[ 0 0 5 4 ] #tfc LstStRt J: r> X . b'^XJiflB^f-' 
ftKSifrf 6 b'-f * * -f A =y A VX-\$ . ^>(A7^y 

tkzti&zktfX'Z. &mcommtfmft2ti&. ztit> 
comma, ^comm^mn-r^Mzmm^titzryv 

M itt'^ y 77 -y T-7 -f f H -7 ( 04*. ^9 U «y 7 # -y 
T-T-yT-^^ yF>!7) £>KS3?-fS Z biz X 

^flSb'x:*? 'J >yTT-«l^-ra. 
[0 0 5 5] ^U7^f^7h'W*-^7b (X<3 

xtt-^-v htco^imzt'TXizmmt&mmz-? 



•y b^^-r h KMZ , 3tX(ifl»!<7)fl8#g«^ -y VyVtm 

JR<o*U>. BHWMa-X-Mi, awwcjeesiifc*' 

^MHl/Cj^ftLfc^'J ^ y HcJOgB^CMJE-rSb' 

«r* T^'^^-fbSiojtb'T-'jrxjiffi^x-^ i>m$-t& 
v—j*) t,ztmLxi> b'x^fiaii, m<r>~fu 
v?70 7izX'?xm®ztL&zbtfxz&. 
[0056] mnfmmi. trximmffiLtzji 

fVt: 9 A A 5 -i VOT H l^X* y h (att^T 4 i 
-C&i.. ff^* (ID) ^'b'TJt-OBfS 

( mx\±m\ >xumco7 v ? 

itLXi>X\>\ 

[0057] McomMwmmx'H, y^mt-<k7c 
TYuxtr-^v y 9 o oco-^emfacommzm^-t 

/' , J77Hl/X*-'<7 hffiBJil^U^WWST- 
S«3*l-, ^ c7)»ffit±-JRWt^- x±f 15 1 / 1 6 W 

jyfy^-x (7>jyb77h) Srffiffl^s*^ fl&(7) 
H*feO«?JISJi* t VP h 9 A 7 <r> tij^j S-f flffl L x X \ \ m 

uf, 3-—nz£ixig&zti& s-hRvrvyft^ 
ttzm^^&niz+ttZMmmzirt&iyvj,-?? 

[0058] m^twmmtmmx'iz. mtfrnrnzti. 

7-h^7y*i?b (Xbo-^, Xf77) <T)&m<?> 
~<50c*-^-y bcr,x- Y&W.£ b'-f ^OB$ Htcv -y b 

[«1] 



X-Y&g 


=t—9X h'J-ilD 


(^-f5>y&B) | 


(I, 4.5) 


Hftl 


2252 


(6.5. 4) 




I. 1 


(4.25, 4) 




1. 2 



S-X-YfigJiX^-yT, 4y?Xhn-?XimcO* 

[0059] mux ittw^-tK . *^s^x 
? htcMii-rsbx^fifBtr^-fex-rs^cT)^^ 

^rV^. Mt-li. Xt779 1 0(craat--Sb7-'^-ii. ^ 



7?A9l5(;j;-5t^§fil.J;7{:(0 1 o)~ 
(2, 0)t(0, 5. 5 ) ~ ( 2 , 5. 5) ^SEffl* 1 

f»7m?^. -o-?xho-?9 2 o^Kii-r 

4b*7**«» 9 2 5T^Six4«#[*|C0^'j7?:^it-r 

3 o izmm-tz bWii 935 7 
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a. 

[00 60] fHJ±7 4>cr>tg>'&tmmz, /'J7 3- 
h'£ t'T#XJifa<7)T-?X h V-AcD^HzfflmZlt 
tzmi. K*>±*Wy7=J-f*fc LTfgfc*l£;rt&a>Xte 

[0061] «*>*1Ni, t'f^ h >J -A&tfSj^l 
(XJitT^co^-fe y M fcx-YftBtcioTtHR? 

[0062] h'^:*?){£g£ . *t»T H PX4H&5^ 
tU> Xmy?-0-7i>l<lt4 V9-V-X- 

<, JlC7)^i, ^hh(Hecht)(Ci:&"Parallel Propa 
gating Embedded Binary Sequences For Parameterizin 
g Two Dimensional Image Domain Code Patterns in Tw 
o DimensionalSpace"fcV^:?-f b JU<7)%mftgtf&5,82 

5,933-t (±&t Lx&mtz^x&mmwzmm-r 

[ 0 0 6 3 ] m 1 Hi, #S&«U!£*<y r- ^-^at^TiS 

S.SIfA'^fxioooii. ytf» 
A£*t#-r.&*g*rtT , ^'fxi o i o*-£tf. v-^-y 

O^"-7^1020ti, y^irXJifftcO^MT^r^ 

ioo oai/t-^rn^A i o 2 oti-^n-efu-y 

b-7-^1030 mtlf. ^fy^-*7b)(I«I$ 
ftTV>.g>. J--if{i, Jaiflj£2E*;fr— XAl 0 SORTS 

OSO^^TtS. jH#^^-^A10 5 0(i, (104 
0 Sr^-LT ) *-y 10 3 0C:JggH-.g>f|2tSy\- 

V y^f 1^ b"<— Xcryf) 

HSW&SIJfccOJgJg-m, *-y 1 0 3 

o^o&ii 104 o<±iif3£tt i o 3 5 izx ~>xmnz 

[0 0 64] 2fs^BJ<7)l 0 7 0A>£><7)ffJ7Jki:, fl^cOX 
^rXtey-F&l/^^£^y:7£-S-tf. ^tr^' 
4 X^*-XA 1 0 6 0 Srfflffl-f S a.—- «4. &&<?>S 



-b/xi- vTizmm-t&rvy&miRL. wrvy 
««y^-XAi 0 5 0{4^iEL^^y7 

3-F£M*3X0, ®W6r^ y-fe-^-7^-V-yf--f 
>?"U -TO.* *-fe— 5*4** h V-7 1 0 3 0KI^ 
nt-^ra/5A 10 2 OT^ltBl^il^ . -tfwtr 

D/5Aii. ^acL^^y7^jE-ri>, ma?'*-** 
i o l oKfgft^ftfcr-^xh y-AowaSMMcr 

^•fex-f-l.. T?-fcXgfrJHi*-y r- y-?l 0 3 0£tf- 
LXmn**-*^l 0 5 OfcrftiMSfU t -f XTW 

7-v\Mxio55 (3ye A -^-x?y-yxiiS* 
[0065] ««^^-xa iooo, *r-j^-yny 
xm. i o o o t m tmM**-XMzmwztix^x i> 

<k^U *•■/ 1 0 3 0X«i-ffilc0r^MX$r^-L 

mnx*}=rXi±mf*v w-? 1 o 

3 0<0&M<i, <7>f^-FXfi«M, 7^^d>>i- 
7\ •feyW^-XtiffifiOv'Xf-A^^tfV'f-Vkxafiv- 
X-rA<y)<5iix"Ct iv>. 

[0066] 105 O&Z/x -f XTP-f 

x l o 5 5 . /umm.'r'U xmf^f < x 
7l/-(^*^XA. <f 7 >y T F yrny 

fctfxDX -V ^ L < \±<&<F>* fj -XA £#tffiES«* 
^rA^xco^fcffioTiJ:^. «SCry^xi0 5 

oavf-fxxi/'f i o 5 5fi, *w^uffs»m, ie 

h b«p&mmizffiz_ t>tix j; 
[oo67] hhm&<mmx\i. ®et/^x^*- 

XA10 6 0*<fl/ t'StX/X<± i7fU t'f^ <>f X 
r^HS y -Xliffi*, i£SXtt4s«i!Ht= <t 

oTffi^iisiTJtiy'y 7*^-sffm«7)^-f ynmwz 
^-bx-rs. 

[0068] *^HHJ4, 7SW&flJfflUT7-y >-h 

L7ty-b2stt/-<r-7y-*^t'f r 5rt:T^-bx-rs 0 $ 
&.WRi/mm<7>*y-t y f ^^y 7fc«^tr*»&<o 

[0069] *»BHti, ^'J7^ffifflLT»-^/-h# 
^xy^X (Mi.(f, y-h;p-y^)*^K#L^y- 
h COT y > h T >7 b t&>^ h'f^T^ -fe X Sr Sffi t , 

immzmmniiz . 

■ royrvuxtr-^v hzmmLxt'T*^?^ a 
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b'r*r?b.x£*iPrf&. 

• J-b^-WZiotz^X^&jtTYVXij-^ h£ 
x'fcT y y h-tZ Zkizi-o . 

«i. m4izffm-t&£?iz, 

■ VV7Th'\sZ.*>-'<.>vh£imLXK:7 : *co?JJ* 
£>c9b'-r:tr ? *X<,zm®C?Z>mfrt>cr) b'-r;*T? b* 

■ ^^)~1~ i JI,Ziotz^XZ:d5tTY\yXii~1^ b 
£<3OTL. ZtllZ£r,X*-7l'-J±frt>t7-'*l l ZT? 

4 . £ iX fc «£ -> T fc' 7* sT V -X&tmigcO:* 

&m tr>j yizftmmixhtjsb. ff*y-^5^ 

[0 0 7 2] jlSMSJUfc^T. a.— *f(i:M A^-f >- 
i: *y>'x 7 b ^-x<7)T^ bx i: ott^-^b^-^jfftr 
z\ktfftfr~>tz. $<bfc. b'x*fc7*?bxr.g>fc#5fc 
tt*flffl-*-&£fc*«»83*U b*7W"?y-fciH« 

b'x*7 l/-A7'J > h*^cOT KU-^Srovsx 

[0073] 3 > bi-?J$»#fc«:BJ!^T-&6 <fc 3 
fc. ^*co?ilfflX{i$fflr^^/l'3>'b^-^X{i3is|S 

[ 0 0 7 4 ] V 7 b^xT*r«#fcBJte J: a 

fc , #IW^<Oifcf:fc«-?V . W5: y7h">i73- 
t -f v^^^^rn^^^-^ioTSmfc^ii^-s 
<Ii:#T£&. SSg#fcBJtea>T&.g>J; 3fc. 
-S' a J^1*jeaH*lil»**flN-* vlfcfci-sT, Xteffii 
*<0ttJSHl«K^S«J'3r* x h v-^offlSSSifc,}: o 

[0075] ^Bflco^^M^^H^-r 



#fcLT«;. «Tfc|SR5gL3:^A*. 7n7t-fa 
3Ef-f^?, DVD. CD-ROM. 7^oH5 

7. ROM. RAM. E PROM. EEPROM. DR 
AM. VRAM. 757^^'JfAMX, 
ft#-b\ (ttlrJ^V I C£#tf) TVxXxiOUi 
•^yXN77;/3 >Rtf/X{±x-?c0fg**IfciS Lfctt 

[0076] #^Xii«Scc03ybx-^MJRTirtS««c 

bA-^c^v-h'^xTXtire-f ^nro-fe.ylfS:$iJffll 

>y-9-* t AI3<7)J--ifX*i*^ir)^^ifij^-f-6ft!i^^ 
^nXi.t^ffS-C^Sy^b^xTSr^tfo .I<y)J:3&- 
y 7 h >7 x T i LT ti . OTfcRBJS L&v . ft M * 
*^U—T-<r y/«>XfA, R3fa-—VT7 

^xT£-£tf. 

[ 0 0 7 7 ] im/^fflrJ^b^-^XJi^ ?D7D 
^7tW7iy/ (V7b7x7) fc^flS^ 

{±. ^Bg^ife^^iiM-rsy^h^xr^^jL-;^' 

$> 0 . KTFfcKBJg L3r v < T * b V -J±CDim 

msmmitf. m^y-bwm^ob^^^^^ 

n-PV)!/^. X7-f b'$mcr>im. *-7\s-Jx 

$r^'j7fiL»=fflM$-frSmco^. *»Bj<0»afcJ: 

[0078] *?fcBJ§<^ifc<7)^jg&tf>Ky x-i^ 3 
±IScoit^fcJ:o-c-irflgTS)4i b\me>frXh&. L 
fc*«-5T. ^Bj^ts^^HWT'. *BH|B#fc*f*fi<J 

izm^tztmnttx^wnzmmx'* *ia»? 

[HI ] n&WfXnULfU X<nx? U — VS/ g -y 
[H2] ^y^-^av-bjL-^T'Uff-tSy-h^ 

[H3] b*T^r. *«7-f h#-KA*aw3-— ras?« 

3 -y h£^-f0-C$)£„ 

[S4 ] *^Bg^gpt'T5r-9"7U-OM$r^lllT'* 

[05] *2&BJ|(7) t'T-'sf-^-? 'J - £»T S fc^C7)lif? 
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[06 ] *mfrnm&mi<m®mm3\w j t<7M 

[09] V^f-^T^TT-^hU-AOiK'f ybfc 
■&0T'$>?>. 

[010] mfRVtt-yis-JxRtss-bftf&^XT- 
[01] 




125 ) 30 1 35 140 14S 



[01 1 ] *»!H*flyflLT*«y ^-^i^W* 

m$tli>^~ H *) x TfftitcDT'Q -y ? 0T-* s . 
[«T#<0lttBJ] 
350. 840 
615. 625. 635 
80 0 f 7^-9"? 'J - 
82 0 /^TKI/X^'yh 
9 0 0 Z.dOtT V isX*}-**. y h 

[05] 



A* A* 
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[02] 



[06] 



220 



215 




200 



-210 



215 



•230 



510 











520 












530 




540 



[03] 



320 



370 




>310 



315 



[08] 



( Ef5 ) 



700 



710 



720 



730 



740 
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[04] 



E 









o 




m 




□□□ 




K 


¥ 

D 


□ 






CD 

□— 


ad 



l^VV^rt^^AiWft /•AfA^aA^^Aa^^ /(AtaMJiAfaf^l 



[07] 
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BACKGROUND OF THE TKVEMTTOM 



Field of the invention 



This invention relates to access and finding a position 
in video or other recording media. The invencion is more 
particularly related referencing a position within recorded 
media using a reference marker related to the position. A note, 
portion of the media itself, or other item related to the 
position may be utilized as the reference -marker . The 
invention is still further related to the use of glyph encoding 
for associating the reference marker to the corresponding 
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position in the recorded media. 

Diac.uaal.ftQ of the background 

Multimedia note -taking systems typically capture audio 
and video during a meeting and slides are created from the 
captured material. (For example, Tivoli, a system running on 
LiveBoard. see Moran, T.P., Palen, L . , Harrison, s_, chiu, P., 
Klmber, D., Ninneman, S. t vanMelle, W., and Zellweger, p. -1*11 
get that off the audio": a case study of salvaging multimedia 
meeting records. Proceedings of CHI '97 cm. Hew York, 
pp. 202-209). Tivoli is designed to support working meetings 
rather than presentation meetings . The ink strokes in Tivoli, 
vhich are indexed to the audio, along with any prepared material 
on the Tivoli slides become the group notes to the meeting. A 
participant using a laptop may "beam" typed text comments onto 
a slide in Tivoli. 

In a similar example. Classroom 2000 , Images of 
presentation slides and audio are captured, but video Is not 
used (see Abowd, G.D., Atkeson, C.G., Brotherton, J . w Enqrvist, 
T., Gulley, P., and LeMon, J. Investigating the capture, 
integration and access problem of ubiquitous computing in an 
educational setting. Proceedings of the CHI '58 Conference . ACH, 
New York, pp. 440-447; andAbowd, G.D., Atkeson, C.C., Feins teln, 
A., ttmelo, C, Kooper. R., Long. S., Sawhney. N.. and Tan i, M. f 
Teaching and learning as multimedia authoring: the classroom 
2000 project, Proceedings of the ACM Multimedia '96 Conference. 
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ACM, New York, pp. 167-198). In addition. Classroom 2000 
requires effort by the presenter to prepare the slides in a 
standard graphics format. The slides are displayed on a 
LiveBoard and note -taking is done with PDA devices pre-loaded 
with slides. These notes are later synchronised to the audio 
and the slides which have been annotated by the professor 
lecturing in front of the LiveBoard. 

In yet another example, the Forum < see Isaacs, E.A., Morris, 
T., and Rodriguez, T.K. A forum for supporting interactive 
presentations to distributed audiences . Proceedings of CSCW ' 94 . 
ACM, New York, pp. 405-41G), is a system uses video as a means 
for distributed presentations. Everyone, including the 
speaker, sits in front of a workstation during a presentation. 
Slides have to be prepared in a specified format. The slides 
can be annotated with te*t and marks drawn with a mouse, l>ut 
the video images cannot be annotated. 

In another example, the STEBAM5 (see Cruz, G., and Hill, 
fc. Capturing and playing multimedia events with STREAMS. 
Proceedings of the Am Multimedia '94 Conference . ACM. New York, 
pp. 193-200), ia a system for presentation capture that uses 
video from room cameras. These cameras are also used to capture 
any presentation content on display. This method has problems 
when activity in the room obscures the display. Note- taking 
during the presentation Is not supported, although the captured 
video streams can be annotated during review by adding text 
comments. Hone of these systems allow interactive integration 
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of live images from cameras and presentation material, into the 
notes . 

In addition, there are also several known atand alone ink 
and audio note-taking systems. For example, FXPAL Dynomite 
(see Wilcox, L. D., Schilit, B . N . , and Savhney. N. Dynomice: 
A Dynamically Organized Ink: and Audio Notebook, Proceedings of 
CHI '97. ACM, New Tork, PD . 186193); and Audio Notebook (see 
StiEelman, I*, me Audio Notebook; Paper and Pen Interaction with 
Szruccuxed Speech. Ph.D. Thesis. MIT. 1997), which uses paper 
with audio recording. Filochat < see Whittaker , s,, Hyland, P., 
and Wiley, M. Filochat: handwritten notes provide access to 
recorded conversations. Proceedings of CHI '94. ACH, Hew York, 
pp. 271-276) , is a PC computer with a pen tablet in which audio 
is indexed with handwritten notes r and NoTime (see Lamming, M. , 
and Newman. W. Activity-based information technology in support 
of personal memory. Technical Report epc-1991 - 103 , Rank Xerox, 
EuroPARC, 1991), was designed to key the user's ink strokes to 
recorded audio or video. 
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Also known are video annotation ays terns. Marquee (see 
Weber, K. , and Poon, A. Marquee: a tooL for real-time video 
logging. Proceedings of CHI '9*. ACM # Hev York, pp. 58- 64) is 
a pen-based system for making annotations while watching a 
videotape . A later version of Marquee has notifications to take 
times tanps on digital video streams from che wherewerewe 
multimedia system (see Minneman, S., Harrison. S., Janssen. B. r 
Kurtenbach, Q. , Koran, T.. Smith. I. r and van Melle. B . A 
confederation of tools for capturing and accessing 
collaborative activity. Proceedings of the ACM Multimedia '95 
Conference. ACM, Nev York, pp. 523-534). Vanna (see 

Harrison. B-, Baecker, R. M . Designing video annotation and 
analysis systems, Graphics Interface '92. Morgan -Kaufmann, pp. 
157-166) ; and BVA (see NacKay, W. E . EVA: An experimental video 
annotator for symbolic analysis of video data, sracniBul letln, 
21 (2), 69-71. 1989. ACM Press) are text b&secl systems, 
VideoNoter (Trigg, R . Computer support for transcribing 
recorded activity, SIGCUI Bulletin, 21 (2), 68-71. 1989. ACM 
Press) displays and synchronizes different streams of activity 
(video, figures whiteboard drawings, text) , but requires 
post-production to transcribe text from the audio or extract 
drawings from a whiteboard. These systems are limited by their 
design baaed on using videotapes rather than digital video. None 
of these systems allow interactive integration of video images 
into the notes. Sharp Zaurus fZaurus Operation Manual, sharp 
Corporation, 1996) is a commercial product, which is a PDA with 
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a digital camera attached. Digital photos can be taken and linked 
to handwritten notes. 

Furthermore, many systems include video playback 
mechanisms. Fig. 1 illustrates a conventional video playback 
device 100 that includes a video display area 102. A timeline 
10 5 shows a current position of the video being displayed, and 
standard VCR controls, including play 115, Fast Rewind 120. Fast 
Forward 125. Step Back 130 , Step Forward 13 5, etc. . are provided 
that allow a user to navigate the video. However, difficulties 
arise in finding specific positions within a video, 
particularly in reference to notes or other items related to 
the video. 

SUMMARY OF THE IHYBNTIQW 
The present inventors have realized that the note- taking 
systems will have increased ease of playback and greater 
utilization if important parts of a video or other data stream 
monitored during notetaking is provided with a simple and 
accurate retrieval mechanism . Roughly described, the present 
invention provides a multimedia note -taking system that allows 
a user to select keyframes or make annotations during a 
notetaking session (meeting, presentation, or other activity) . 
Each keyframe and/or annotation is captured and indexed to a 
position of a video of the note taking session or other data 
stream related to the notetaking. The captured keyframes and 
annotations are printed on a paper sheet with glyph encoding. 
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and the printed sheet la used to access the important parts of 
the video via scanning the glyph codes associated with a target 
keyframe or annotation and referencing the important part of 
the video or data stream indexed by the target keyframe or 
annotation. 

These and other objects are accomplished by a system for 
note- taking with digital video and ink (also referred to as 
Notebook) , The invention may be embodied in many ways, 
Including , but not limited to any of the following. 

An interface for accessing a data stream, comprising, an 
output, comprising, at least one object that corresponds to at 
least a portion of said data stream, and a code associated with 
each object that identifies a portion of said data stream 
corresponding to the object. 

A method of making an index for access to a data stream, 
comprising, identifying at least one ob j ect related to said 

data stream, determining an offset into said data stream 
corresponding to each related object, and creating an index that 
identifies each offset and corresponding related object. 

A method of accessing a data stream, comprising, 
scanning an object corresponding to said data stream, 
determining an offset into said data stream based on the object 
scanned, and accessing a portion of the data stream 
corresponding to the offset determined. 
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And, a device for accessing video from notes, comprising, 
a scanning device configured to scan glyph codes embedded in 
a video index, a computing device configured to read said scanned 
glyph codes and access a portion of said video based on the 
glyph codes, and 

a display mechanism configured to output the retrieved video 
portion . 

The invention also includes a method whereby any of 
multiple data streams 1 input into a note- taking device may have 
portions or frames captured for later reference, and may allow 
a user to make annotations related to any of the data streams 
captured (such data streams including any of user inputs 
including pen-based inputs, a video input, multiple video inputs , 
data streams, whiteboard inputs, etc.). The method includes 
making an annotation or selecting a frame from one of the data 
stream's input, saving and indexing the annotation or selection, 
and encoding and attaching a glyph that one of identifies the 
saved selection or annotation. 

The steps are repeated for any of multiple selections 
and/or annotations made during a presentation, ox data 
collection effort . The annotations and selections and attached 
data glyphs may then be printed out on a paper or placed on other 
electronic media for selection to initiate a playback of the 
saved selection or annotation, or begin playing back: any of the 
data streans at a point associated urith the annotation or 
selection and associated glyph. A scanner may be utilized to 
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scan a data glyph, and data scanned fron the glyph ia utilized 
to identify and retrieve the video and an attached (or other) 
device starts playing the video or other data stream fron a 
location of the data stream identified by the selection and/or 
annotation associated with the scanned glyph. 

The invention may be practiced automatically whereby the 
data streams are captured and automatically summarized by 
Identifying key frames or other data selections which are then 
extracted and embedded with glyphs identifying a location in 
the data stream where the selection was extracted. Glyph 
referenced annotations and/or selections made by a user manually 
may also be made separately or included in an output of the 
selected key frames. 

The invention may be practiced over a network, such as the 
Internet, where a device captures a downloaded or broadcast 
video stream for annotation or note- taking. Outputs of the 
invention, such as paper output having a selection and 
associated glyph, may be scanned by a user, the glyph then 
identifying an http or other network-related address to a server 
that then begins streaming the video from the position indicated 
by the glyph scanned and output on a terminal such as a PC, palm 
device, web TV-based, or other display mechanisms. 

Various embodiments of outputs of the present invention 
are described, including block glyphs associated with 
annotations or images on printed pages, an address carpet of 
glyphs may be provided in conjunction with an annotated timeline, 
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and a two- dimensional address carpet with X-Tf positions mapped 
to tine and provide selected access to video access similar to 
a retrieval interface of any device, or a summary interface on 
a web browser- type product. 

BRIEF DESCRIPTION OF THE PRA KINGS 
A more complete appreciation of the invention and many of 
the attendant advantages thereof will be readily obtained as 
the same becomes better understood by reference to the following 
detailed description when considered in connection with the 
accompanying drawing s , wherein: 

Fig. 1 is a screenshot of a conventional video playback 
device; 

Fig. 2 is a diagram illustrating a note - taking application 
client running on a pen -based computer; 

Fig. 3a is a screenshot of a note - taking - type application 
having video and whiteboard inputs, and a user annotation area; 

Fig. 3B is an example of a Manga video summary according 
to the present invention; 

Fig. 4 is a block diagram of one embodiment of an 
annotation/da ta - segment capture device and information flow to 
produce video summaries according to the present invention? 

Fig. 5 is a flowchart that describes a process for 
capturing and indexing selections and annotations according to 
the present invention; 

Fig. 6 is an example output of glyphed annotations and an 
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indexed video segment according to the present invention; 

Fig. 7 is a flowchart that describes a process of 
summarizing a video and embedding glyphs according to the 
present invention; 

Pig. 8 is an illustration of a keyframe summary (including 
video and whiteboard snaps) having an attached timeline and 
associated data glyph carpet for accessing points in a 
multi-Media data stream; 

Pig. 9 is an illustration of an example data carpet mapping 
multiple points of selected keyframes and annotations captured 
in a note- taking system to a multimedia data stream; and 

Fig. 10 is a block diagram of a hardware configuration 
being utilized to access points in a video stream over 
network -connected devices using the present invention. 
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DESgRIPTTOW OF TH E PBRPERRgD EMBODIMENTS 

Referring again to the drawings, wherein like reference 
numerals designate identical or corresponding parts, and more 
particularly to Fig . 2 thereof, there ia illustrated a pen-based 
computer 200 running a note -taking application 210. The 
note -taking application includes a main area 215 resembling a 
paper notebook page for writing, capturing and, annotating 
Images . There is a small video window 2 20 for viewing an active 
video. The active video comes from an active channel, and the 
user may change channels to view other video streams or present 
another type of data stream to the user or application. The 
note- talcing application includes facilities to grab a frame that 
is showing in the video window and place the captured frame (225. 
for example) in a margin of the main area, or grab a frame as 
set it as a large background image. 'Ink* strokes of a user's 
pen device 23 0 are also captured. The thumbnails, background 
images, and Ink strokes are timestamped and are utilized to 
provide an index into the video or other data streams. 

The active video may be provided to the note- taking 
application via any source. The video source can be captured 
in a variety of ways: from a room camera or document camera, 
from a tap into a rear projector. TV, a document camera for 
overhead slides, VCR or any video stream, or from a small 
portable camera attached to a pen computer, or any other type 
of data input device. Multiple video or other data sources may 
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be provided to the note- taking system. 

Fig. 3 illustrates a screenshot 300 of a note- taking system, 
including a video window 320 and video controls 310. Next to 
the video window are three buttons for interacting with the 
video: the top button (channel changer button 322) changes 
channels r the middle button 324 snaps a thumbnail (thumbnail 
370, for example] into the margin of a note page area 31$, and 
the bottom button 326 snaps in a large image 390 (i.e. , an image 
available for annotation or perhaps enlarged for viewing) into 
the note page area 380. 

Farther to the left of the video window 320 is a set of 
VCR-type controls 360. A timeline 350 has a pointer 355 for 
Indicating the current time position of a video being played 
back. The pointer may also indicate a time position of multiple 
data streams being captured during a note -taking session (for 
example, each of a video input, whiteboard input, and pen-based 
inputs from a user) . The timeline can also display indexing 
information to facilitate browsing and reviewing. The exact 
configuration of the note- taking device may be of any form and 
different or additional control mechanisms other than as 
described herein may be provided. 

The system allows users to take notes on the pen- based 
computer while viewing video that is being digitized and stored 
on a server (or stored locally or on other devices r e.g. tape, 
cd, etc.). The notes are time-stamped and synchronized with 
the video, in addition, the user can snap still images from 
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the video into the notes, either as thumbnails in the margin 
or as full -size images in the note page. These images are also 
time-stamped and synchronized with the video, so that they too 
serve as indexes. 

After a note- caking session, the note- talcing system can 
be used to play back the recorded video. The playback position 
can be controlled by standard VCR controls such as play forward 
or reverse. In addition, specific parts of the video can be 
accessed using the indexes created by the pen strokes and still 
images in the notes. The user selects the desired index object 
and presses the PLAY button (or invokes another play mechanism, 
e.g. Double Click, etc. I . An annotated timeline is also 
available (see Fig. 1) to position the time of video playback. 

The above note- taking system roughly describes the 
NoteLook system developed at FX Palo Alto Laboratory. The 
NoteLook and other note- taking systems may be utilized to 
capture images, snaps, annotations, etc., upon which to apply 
the present invention. Many other types of devices supporting 
any of capture, annotation, enhancement, or other processes on 
any of video, audio, multimedia and other data types may also 
be utilized to apply the present invention. All that is needed 
is a device capable of capturing any type o£ data stream and 
facilities (included in the device or otherwise accessiblel for 
applying the methods and/or processes of the present invention. 

It is preferred that frame rate compression be utilized 
to save bandwidth. Automatic change detection (slide change 
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detection, for example) may be utilized and only changes ia a 
media stream are sent to the note- taking device . Alternatively, 
a frame per second rate of material in the media stream may be 
reduced (i.e.. a video shot at 30 fps may be reduced to 1 or 
2 fps) . This may result in a jittery display at the NoteLook 
client, however, a perfect video at the note- taxing device is 
not essential to the note- taking process, and. in fact, not 
needed in situations where the notetaker is present at the video 
(because the notetaker can hear and see what is taking place} . 
The note- taking device may store the video/data stream or the 
video/data stream may be captured or maintained on a separate 
server. 

Note-taking systems are particularly useful in a 
conference room setting, where video from room cameras and 
projectors can be transmitted to the NoteLook computer over a 
wired or wireless network, while it is easy to imagine such 
note* taking devices being available in a conference room, it 
is less likely that users will have their own devices for use 
outside the room. Thus it is important to give users an 
alternate means to access their notes and the recorded video 
outside the conference room. One of the easiest ways to do this 
is to provide the user with a printout of the notes. However, 
this alone does not give access to the stored video. 

The present invention embeds data glyphs in any of notes, 
annotations, clips, snapshots (all collectively referred to as 
notes) . onto printouts or other outputs that provide access to 
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the video data stream that vas utilized during a note- taking 
session. The glyphs provide time offsets into the video so that 
a specific section of video can be accessed. In addition, the 
glyphs can encode a location of the recorded video associated 
with these notes. 

Another application in which glyphs can be used to provide 
access to video f ron paper is video summaries . One type of video 
summary suitable for printing is the Manga -style summary 

described in Uchihashi et al . , U.S. Patent Mo. „ , 

entitled "AUTOMATIC VIDEO SUMMARIZATION USING A MEASURE OF SHOT 
IMPORTANCE AND A FRAME - PACKING METHOD, " Serial No. 0 9/276, 539 , 
f Had March 12, 199 9, which i s incorporated herein by reference , 
in its entirety. Here, a video is automatically analyzed and 
keyframes of varying sited are selected for the summary. Pig. 
3B shows an example video summary. The larger keyframes 
correspond to more important parts of the video. Summaries can 
be placed on Web pages where they provide access to the video, 
when a keyframe ia selected, the video begins playing from that 
f r ame . 

In one embodiment, the summaries are printed with embedded 
glyphs. This allows access to video from paper, rather than 
through a computer interface on the Web. The user can use a 
hand scanner to scan a glyph embedded in a selected part of the 
printed video summary. The scanner is connected to a device 
ouch as a Web TV or to a regular PC which then accesses the portion 
of the video/data stream indexed by che scanned glyph. 
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The embedded data glyphs in printed notes or in video 
summaries enable a user interface to the video, video is 
accessed by selection of a note or an image captured from the 
video having an embedded glyph that references a position in 
the video, a segment of the video, or a set or at least one 
multimedia or other data streams. 

Fig. 4 illustrates a device according to the present 
invention. A capture device 400 includes at least one input 
stream of data 410 , which may be any of audio, video, whiteboard, 
or other data inputs. User inputs 420 are utilized to identify 
portions of any of the data input streams 410 and to make 
annotations and/or notes associated with any of the input 
streams or other events of which notes are taken by the user. 

The capture device 400 includes a capture/annotation 
mechanism 425 that captures user - selected portions of the input 
data streams 410 and any annotations or other notes made by the 
user, provides those user- selec ted portions, annotations and/or 
notes with on ID, index, or tlmestamp identifying a position 
in the input data stream, and stores them to a storage device 
43 0. The capture device 400 also includes channels for 
forwarding che input data streams to the storage device 430 for 
storage. (Alternatively, input data streams may be provided 
independently to the storage device 430, or simply maintained 
on a server.) 

A summary mechanism 440 retrieves the captured snapshots, 
annotations, and notes and formats them in a appropriate format 
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for output. In addition, the summary device 440 might provide 
independent summarization and keyframe selection of any one or 
more of the input data streams 410. The output is formatted 
Cor any of a web display such as on a brovser device on a personal 
computer 450. printed out in hard copy form (paper) 455, or 
stored on an electronic format (floppy disc, for example) 460 . 
The outputs may be provided to any other type of output device 
including other web -based products, web IV for example, etc. 

Fig. 5 illustrates a process according to the present 
invention. At step 500, a note - taking eystem is started. The 
note* taking system Includes any one or more of the features as 
presented above, or is any type of device that receives a data 
stream that may either be commented, annotated or selected by 
a user. At step 510, a user of a note- taking system makes an 
annotation or selects portions of data provided to the 
note-taking system. At step 520, any annotations or selections 
made are saved and indexed in a manner that allows retrieval 
of the selection and/or annotation, and identifies a position 
in at least one of the data streams provided to the note- taking 
system. An annotation might also be made in reference to a data 
stream that is not provided to the note- taking system, but played 
in conjunction with a note- taking system. 

At step 530 a glyph is encoded with information identifying 
either the annotation or selection and a position in a specific 
one or multiple data streams, and glyph is attached (or 
maintained with reference) to the selection or annotation. At 
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step S40 the annotation/ selection process and glyph attachment 
ia repeated. 

As will be appreciated by those skilled in the art, based 
on the present disclosure, many different variations of the 
flowchart of Fig. S may be implemented to practice the present 
invention. For example, annotations and selections may be all 
made and stored in a loop prior co encoding each of those 
selections and annotations in a batch format. Any of the 
processes may be performed in parallel or serial depending on 
architecture and programming techniques. What is important is 
that annotations and selections are made and glyphs are 
associated with each of the annotations and/or selections r the 
glyphs containing information or pointing to information that 
Identifies a portion of a data stream corresponding to the 
annotations and/or selections. 

The glyphs may contain information on the location of the 
associated video on the server (corresponding to the particular 
note set) , or time offsets into that video, or the glyphs may 
contain pointers to any other type of picture, video, audio, 
document, or data, rt makes the most sense to use the glyphs 
to link to a video being displayed at the time a selection, or 
annotation was made, or to link the glyph to a document being 
annotated or commented on at that time. Alternately, the user 
could manually specify the location of the video (via a pop-up 
window or other mechanism provided by the note -taking device) . 
Once output onto paper notes, using a scanning device 
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(hand-held scanner, for example) t the user scans image areas 
is the paper notes, thus designating the desired selection. 
Glyphs in the scanned image provide appropriate information to 
initiate playback of the video at the particular time offset. 

Various embodiments of glyphs are utilized by the present 
invention. In one embodiment, a block glyph is provided for 
each indexed object in a printout or other output, in the 
example in Fig. 6, there is a glyph 615 associated with the 
thumbnail (snap) 610, and with each group of ink strokes (Ink 
strokes 620. 630 r and respective glyphs 625, 6351 on a page of 
a note -taking device. The objects (thumbnail and ink strokes) 
were indexed and attached to the respective glyphs according 
to the process in Pig. 5, or an equivalent. Other types of 
objects may be similarly indexed and provided with a glyph 
including any image or data, including background images. 

in one embodiment, the location of the video (or other 
multimedia /data stream) associated with the object is encoded 
In the attached glyph on the note page. Alternatively, the glyph 
may provide an entry point into a table that identifies the video 
and of rset as described below. 

The ink strokes themselves represent a group of related 
ink strokes- Each individual pen stroke is not indexed into 
the video, but instead, the present invention indexes groups 
of pen strokes. Many methods for grouping pen strokes may be 
Utilized and still fall within the scope of the present invention. 
Grouping via any of sentence structure (grouping complete 
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sentences or paragraphs together using character recognition 
and grammar rules, for example), word recognition (grouping 
words), or time, for example. However, it is preferred that 
pen strokes are grouped by time, so that tii&e differences between 
strokes in the group are smaLl. Thus, an importance of any loss 
of indexing resolution is reduced. The present inventors have 
determined that a 1 sec tine delta is best for grouping pen 
strokes. 

The design also works for video summaries or any 
combination of selections, annotations, and video summaries. 
In the case of video summaries, a block glyph would be associated 
with each keyframe in the summary. 

Pig. 7 is a flowchart of a process □£ summarizing a video 
and embedding glyphs according to one embodiment of the present 
invention. At step 70 0. a video is summarized. The video may 
be any video stream input to a note- taking device or separately 
recorded in conjunction with a conference, meeting, activity, 
or other event, video need not be the object upon which Che 
summarization process acts, for example an audio data stream, 
or other type of product may also be summarized and indexed. 
For products other than those that are normally represented by 
a visual image (audio, for example) , an icon, graph, or other 
representation identifying the product may be incorporated. 

At step 710, keyframes from the summarized video are 
selected. Any selection algorithm may be utilized, and/or a 
set of external criteria way be applied to select the keyframes. 
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At otep 720, the keyframes are indexed into the video or other 
data stream and associated with a glyph that encodes the indexed 
position. Other data nay also be encoded into the glyph, 
including an identification of the video or data stream, an http, 
ftp, or other address where the underlying data stream can be 
accessed. Alternatively, the glyph may be encoded with a 
locator number or other identifier that references a table entry 
where the underlying data stream nay be accessed (i.e., step 
72 0 may include building a reference table) . 

tn another alternative, auto indexing may also be utilized 
to automatically determine important video shots (such as a 
slide change on a whiteboard data stream) that may automatically 
be captured and made into specific notes (using change detection 
criteria, for example) . Such auto Indexed material may also 
be linked to a timeline such as shown in Fig. 8, e.g. slide change 
810 shown on summary and corresponding timeline snapshot 860 
at 25:00. 

At step 730, other annotations or selections are also 
indexed and provided a corresponding glyph. And, at step 74 0, 
an output is produced (paper, for example) from which to 
reference and recall the video and/or data streams indexed by 
the g 1 yph.fi . 

In another embodiment, the embedded data glyphs are 
provided in the form of a glyph address carpet. For example, 
the glyph address carpet may be placed in a horizontal timeline 
underneath the printed notepage or summary. Fig. 8 illustrates 
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a video summary 800 with a glyph address carpet 820 and 
associated timeline 840. The timeline 840 is used to nap 
horizontal position to tine, indexed in the video as encoded 
in a corresponding location of the glyph address carpet 820, 
linearly or nonlinearly, so that the user can access any point 
in the video, regardless of whether there is an Index object 
(from a note) or keyframe (from a summary) for that time. 

As an enhancement, any objects (from the video summary or 
notes) may be displayed at appropriate positions on the timeline 
to aid navigation. Various summary objects (keyframes) 860 are 
shown in Fig. 8, and are placed in a position relative to an 
encoded index in the glyph address carpet 820 corresponding to 
a position of the summary object in the video. 

Positioning of the data glyph carpet need onLy be in 
conjunction with the timeline to allow a user to perceive a 
connection between keyframe objects and the time line, such that 
when an object or place on the timeline is scanned, glyph codes 
corresponding to the object or place on the timeline are also 
scanned. Thus, the position of the glyph address carpet 820 
may be above the timeline as shown in Fig, 8, placed below the 
times line, or integrated with the timeline. 

Annotations and/or coloring on the timeline or glyph 
address carpet may be provided to indicate a linear or nonlinear 
time scale . Multiple parallel address carpet strips can be used 
to access multiple video <or other data) channels with identical 
or distinctive fseparate or different! time scales. This is 
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useful when a notetaking device is utiLized, ouch as NoteLook. 
where multiple video channels way be recorded or snapped from. 
Multiple strips caji also be used to provide extended time scale 
of a video with spreadout selection. 

A spreadout selection allows portions of a video or other 
data stream to he selected and moved or added to other files. 
For example, in a video timeline implementation with a glyph 
data carpet, portions of the timeline (and the embedded glyph) 
may be selected. These selections may be dragged and dropped 
into other applications, saved as separate filed, played, or 
any function performed thereon. These functions may be 
launched by dropping the selection into an icon representing 
an application configured to perform the function, or via 
further selection from a pop- up window (e.g. a right- click 
pop-up window) . for example. The function or application would 
then operate on the video clip represented by the selection. 

For the address carpet in the timeline (or other 
embodiments described herein) . a table or other functional 
mapping may be utilized to map the horizontal position on the 
address carpet to the time associated with the video. This table 
could be stored as a glyph code on the paper, which would be 
scanned and read into a computing mechanism that would then 
reference any portions of the glyph carpet subsequently scanned 
to the read table to determine the index into the video 
corresponding to the portion of the scanned glyph carpet. 
Alternately, the table could be stored on a server (a server 
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that also maintains the digitized video or other data, for 
example) . The location of the video can be provided by a 
separate block glyph. 

And, another solution is to integrate a distributed label 
for the video location into the address carpet in the timeline. 
Por example, both a video name (ID), and a time (position or 
Index) into the video may be encoded into the timeline or other 
glyph. 

in another embodiment, glyph information is embedded in 
a two-dimensional address carpet. Fig. S illustrates one 
embodiment ol a two -dimensional address carpet 90 0. The glyph 
address carpet position can be selected to a precision of one 
glyph, which Is typically about a sixtieth of an inch on the 
page, which is on par with current electronic computer display 
resolutions. Other resolutions using new or improved glyph 
type technologies may also be utilized. Although the primary 
embodiment of the present invention uses a paper interface 
(printouts) with notes and embedded glyphs, alternate 
embodiments may utilize any type of output. Por example, a 
computer display having sufficient resolution to display the 
notes and glyph encodings such that they may be scanned by a 
user. 

In some embodiments, a table is created which maps X-Y 
locations on the two-dimensional carpet in the neighborhood of 
note objects (strokes, snaps) to a time in the video. Table 
1 illustrates an example table for mapping X-Y positions of note 
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objects. 



X-Y position 


data scream id 


Index 

(timing position) 


(1, 4.5) 


videol 


2252 


(6.5. 4) 


inks 


1.1 


(4.25. 4) 


inks 


1.2 



Each X-Y position corresponds to a snap, inkstroke or other 
object. A selection mechanism for accessing video 
corresponding to notes or other objects may determine the note 
or Object closest to a position of the scanned 2 dimensional 

glyph. 

Alternatively, the table might be more comprehensive and 
define a complete range (an area on the 2 dimensional carpet) 
for accessing the video position associated with an obj ect . For 
example, video associated with snap 910 might be accessed from 
from a range of between (0,0) to (2.0) and (0, 5.5) to (2. 5.5) 
as indicated by box 915. Video associated wi th ink strokes 920 
might be accessed via scanning any glyph within a boundary as 
indicated by 925, or video associated with ink strokes 930 by 
scanning any glyph in a boundary indicated by 935. The sice 
and shape of the boundaries may be computed by any method for 
dividing areas relative to the objects to be accessed. 

As in the timeline case, the table correlating glyph codes 
with indexes into video or other data streams can be stored as 
a glyph code on the paper, or other configurations, such as 
stored on a server with the digitized video, in another 
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alternative, the glyph Itself is encoded with all the needed 
information to access the video. 

in yet another table alternative. Instead of referencing 
video streams and index (or offset into the video) by X-y 
position, a code embedded in the glyph is read by a scanning 
device and used to reference the video and any offset or index 
into the video. 

The location of the video can be embedded in the address 
code as an extended address distributed label or as an 
interleaved or interlaced data code. For example, data encoded 
in the glyph such as video ID and position may be encoded in 
alternating sections of the embedded glyph code. Alternatively, 
other methods for scrambling or encoding the data in the glyph 
codes may be utilised, including techniques such as those 
disclosed in Hecht. U.S. Pat. 5.825,933, entitled "Parallel 
Propagating Embedded Binary Sequences For Parameterizing Two 
Dimensional Image Domain Code Patterns in Two Dimensional 
Space, - which is incorporated herein by reference in its 
entirety. The application of the two-dimensional address 
carpet may be similarly applied to is similar for video 
summaries . 

Pig. 10 illustrates one embodiment of hardware for 
applying the present invention in a network environment. A 
computing device 1000 contains a storage device 1010 that 
maintains video and/or data streams captured or played during 
a note- taking session. A server program 1020 is configured to 
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serve tip portions of any of the captured data streams upon 
request from browser or other types of devices. The computing 
device 1000 and server program 102 0 are each connected co a 
network 1030 (Internet, for example) . A user has a computing 
mechanism 1050 with attached peripheral scanning mechanism 106 0 
and display device 1055. The Computing mechanism 1050 may be 
a standard personal computer or a web -TV- based device which is 
connected (via 1040) to the network 1030. In a typical 
embodiment, the connection 1040 to the network 10 30 is performed 
through a telecommunications company 1035, and such connection 
may be any of wired or wireless connections. 

An output from the present invention 1070 includes various 
snaps or notes and corresponding glyphs, a user, utilizing 
scanning device mechanism 10S0, selects and scans a glyph 
associated with a specif ic note/snap. Computing mechanism 1050 
reads the scanned glyph code and formats an appropriate message 
that is sent to network 1030 and received by server program 102D. 
Tiie server program accesses a relevant portion of a data stream 
stored on storage device 1010 that corresponds to the scanned 
glyph. The accessed portion is then transmitted via network 
1030 back to computing mechanism 1050 and displayed on display 
device 1055 (computer screen or standard television set). 

The computing mechanism 1000. server program 1020 and 
storage device 1010 may be located on a same computing; mechanism 
as computing mechanism 1000, or may alternatively be maintained 
on different computing mechanisms also connected via network 
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1030 or other devices. Connections between any of the servers 
and computing mechanisms and network 1030 may be any of wired 
or wireless communications systems, including satellite, 
microwave, cellular, or other systems. 

Computing mechanism 1030 and display device 1055 may take 
the form as a mobile computing device and display mechanism such 
as a laptop computer, Palm- type computer or any electronic 
device that includes a scanner or other mechanism for inputting 
the glyph codes. The computing device 1050 and display 105 5 
may be incorporated into a vehicle such, as an automobile, 
delivery or repair van dashboard, or in a hotel lobby or hall 
adjoining a conference room, or anywhere that the system might 
be utilized. 

in one embodiment, the scanning device mechanism 1060 is 
integrated into a remote control for a TV and/or Web TV device. 
Thus, a user may have a printed summary or article, advertisement, 
or any type of media having embedded glyphs according to the 
present invention and utilize the scanner to access video or 
other data associated with the printed summary, article, etc. 

The present invention utilizes glyph technology to access 
video from printed notes and summaries, in addition, this 
invention includes specific techni<yues for encoding video 
location and tine offset in glyphs, including table and direct 
encoding. 

The present invention uses glyphs to provide access to 
video from printouts of notes captured from an electronic 
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note- taking device (e.g., MoteLook) , including: 

associating block glyphs with groups of handwritten 
strokes and images in a note page; 

• using a glyph address carpet to create a timeline 
for video, thus providing video access from a paper 
timeline; and 

* using a two-dimensional address carpet over the note 
page* thus providing a paper interface that may toe 
operated by a scanner such as a pen -based scanning 
device for accessing video from index objects. The 
paper interface may be produced by printing an output 
such as Fig. a onto a glyph carpet page. 



3B: 



Far printed Manga video summaries, as illustrated in Fig, 

• using a glyph address carpet to create a timeline 
for video, thus providing video access from paper 
analogous to video access from a timeline on the Web; 
and 

using a two-dimensional address carpet over the 
summary page, thus providing a paper interface 
similar to the Web interface for accessing video from 
keyframes . 



The present invention also provides techniques for 

specifying the location of the video on the server and the time 

orfset Into the video. These allow the video source and time 

ofeset to be encoded in the sane glyphs, bo that separate scans 

are not required to specify a video source. The present 

invention includes: 

use of a distributed label in a glyph carpet address 
to identify source and time position in video; and 

use of interlaced code in the address carpet to 
identify source and time position. 



Experimental results have shown that users may prefer a 
combination of the timeline and object based access. Further, 
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the use of paper to access video was well received, and should 
prove useful for video summaries as well. Overprinting of 
NoteLook note pages has been done on glyph carpet pages with 
robust decoding of addresses from handwritten note segments and 
from video frame prints. 

The present invention may be conveniently implemented 
using a conventional general purpose or a specialized digital 
computer or microprocessor programmed according to the 
teachings of the present disclosure, as will be apparent to those 
skilled in the computer art. 

Appropriate software coding can readily be prepared by 
skilled programmers baaed on the teachings of the present 
disclosure, as will be apparent to those skilled in the software 
art. The invention may also be implemented by the preparation 
or application specific integrated circuits or by 
interconnecting an appropriate network of conventional 
component circuits, as will be readily apparent to those skilled 
in. the art. 

The present invention includes a computer program product 
which is a storage medium (media) having instructions stored 
thereon/in which can be used to program a computer to perform 
any of the processes of the present invention. The storage 
medium can include, but is not limited to, any type of disk 
including floppy disks, optical discs, DVD. CD-ROMs, microdrive. 
and magneto. optical disks, ROMs, Rams, BFRONs, BEPROMs, DRAMs, 
VRAMs, flash memory devices, magnetic or optical cards. 
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nanosystems (including molecular memory ICs) , or any type of 
media or device suitable for storing instructions and/or data. 

Stored on any one of the computer readable medium (media) , 
the present invention includes software for controlling both 
the hardware of the general purpose/ specialized computer or 
microprocessor, and for enabling the computer or microprocessor 
to interact with a human user or other mechanism utilizing the 
results of the present invention. Such software may include, 
but Is not limited to, device drivers, operating ayatemg, and 
user applications. Ultimately, such computer readable media 
further includes software for performing the present invention, 
as described above. 

Included in the programming (software) of the 
general/specialized computer or microprocessor are software 
modules for implementing the teachings of the present invention, 
including, but not limited to capturing and annotating media 
streams, producing a timeline of significant note • taking events, 
linking still frames to points in or segments of a media stream 
and to glyph codes representing the same, recognition of slide 
changes, summarization of keyframes, producing tables 
correlating frames, ink strokes, and other data to glyph 
positions, and communication of results according to the 
processes of the present invention. 

Obviously, numerous modifications and variations of the 
present invention are possible in light of the above teachings. 
It ia therefore to be understood that within the scope of the 

appended claims, the invention may be practiced otherwise than 
as specifically described herein. 
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CLAIMS 

What is claimed is: 

1. An interface for accessing a data stream, 
comprising: 

am output, comprising, 

at least one object that corresponds to at least a portion 
of said data stream, and 

a code associated with each object that identifies a 
portion of said data stream corresponding to the object. 

2. The interface according to Claim 1, wherein said 
codes are embedded in each corresponding object. 

3. The interface according to Claim l, wherein said 
output further comprises a table that correlates each of said 
codes to an offset in said data scream. 

4. The interface according to Claim 1, wherein said 
output is a printout. 

5. The interface according to Claim 1. wherein said data 
stream is a video. 



The interface according to Claim 1. wherein said code 
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is a data glyph. 

7. The interface according to Claim 3, wherein said 
table is a data glyph. 

8. The interface according to claim 1, wherein: 
each of said codes comprise a data glyph carpet; and 
said objects are placed in positions corresponding to 

their corresponding codes in the data glyph carpet. 

9. The interface according to claim 8, wherein said data 
glyph carpet is positioned in conjunction with a timeline such 
that codes in said data glyph carpet correspond to points on 
said timeline which also represent an approximate offset into 
said data stream. 

10 . A method of making an index for access to a data stream, 
comprising : 

identifying at least one object related to said data 
stream; 

determining an offset into said data scream corresponding 
to each related object; and 

creating an index that identifies each offset and 
corresponding related object. 

11 . The method according to Claim 10 . further comprising 
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the step of: 

outputting each of the identified objects in association 
with a code that identifies the corresponding offset. 

12. The method according to Claim 10, wherein said data 
stream is a video or multimedia data file. 

13. The method according to Claim 10 , wherein said obj ect 
is at leaat one of a keyframe of said data stream, a selected 
frame of said data stream, and a user annotation made in 
reference to said data stream. 

14 . The method according to Claim 13, further comprising 
the step or: 

grouping sets of related digital ink: strokes as a user 
annotation . 

15. The method according to Claim 13, wherein said step 
of grouping comprises grouping digital ink strokes into sets 
wherein each digital ink stroke in any one group was created 
by a user within a predetermined time delta. 

16. A method of accessing a data stream, comprising: 
scanning an object corresponding to said data stream; 
determining an offset into said data stream based on the 

object scanned; and 
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accessing a portion of the data stream corresponding to 
the offset determined. 

17. The method according to Claim 16. wherein said step 
of determining comprises: 

reading a code associated with the scanned object; 

accessing an offset from a table that correlates the 
associated code to said offset. 

IB. The method according to Claim 16, wherein said step 
of accessing comprises: 

sending a request for a portion of said data stream 
corresponding to said code to a remote computer; and 

receiving said portion of the data stream. 

19. The method according to claim 18, wherein said 
request comprises one of an http, ftp, or other Internet related 
request type; and 

said remote computer is connected to the Internet and has 
access to at least said portion of said data stream. 

20. The method according to Claim 18. wherein: 

said request includes the code read in association with 
said object; and 

said remote computer accesses a table that correlates said 
code to the corresponding portion of said data stream. 
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21. The method according to Claim 16. wherein said step 
of receiving comprises receiving a streaming video. 

22. A device for accessing video from notes, comprising: 
a scanning device configured to scan glyph codes embedded 

in a video index; 

a computing device configured to read said scanned glyph 
codes and access a portion of said video based on the glyph 
codes; and 

a display mechanism configured to output the retrieved 
video portion. . 

23. The device according to Claim 22, wherein: 
said computing device include a network connection 

capable of connecting to any of a wired or wireless network; 
and 

said computing mechanism is further configured to request 
said portion of said video from a remote device via said network 
connection.. 

2A. The device according to Claim 23, wherein said 
computing mechanism includes a table that correlates said glyph 
codes to an offset into said video. 



25. The device according to Claim 24, wherein aaid 
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computing mechanism is further configured to read said table 
via said scanning device. 

26. The device according to Claim 22, wherein said 
computing device comprises a palm type computer and a wireless 
network connection. 

27. The device according to claim 22. wherein said 
scanning device is integrated into a remote control for 
auxiliary equipment. 

28. The device according to Claim 27, wherein said 
auxiliary equipment is one of a computer, a television, and a 
Web -TV device. 
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ABSTRACT 

Recorded video is accessed from printed notes or summaries 
aerived from the video. Summaries nay be created automatically 
by analyzing the recorded video, and annotations are made by 
a user on a device for note- taking with digital ink and video. 
The notes and/or summaries are printed along with data glyphs 
that provide time based indexes or offsets into the recorded 
video. The indexes or offsets are retrieved by scanning the 
glyph on the printout. The glyph information can be embedded 
in the printouts in many ways, one method is to associate block 
glyphs with annotations or images on the printed pages . Another 
method is to provide an address carpet in an annotated timeline. 
Yet another method is to provide a two- dimensional address 
carpet with X - Y position mapped to time which can be used to 
provide selected access to the video. The accessed video may 
be played back on the note -taking device on a pen computer, or 
on a summary interface on a Web browser- type device. 
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(57)Abstract: 

PROBLEM TO BE SOLVED: To provide a video 
monitored during note generation or an important part of 
other data stream with a simple and accurate retrieval 
mechanism. 

SOLUTION: A recorded video is accessed from a printed 
note or summary introduced from the video. Analyzing 
the recorded video automatically generates a summary 
and a user for a device generating a note in digital ink 
and video provides a remark. The note and the summary 
are printed together with a data glyph providing a time- 
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